m 


tom  «i(iMaa«  te  Ml  pipw  an  thoM  or  tfM  aMhdt 
4o  avt  awMmay  MllMt  tha  atm  at  da 
tlMBt  at  Dakam  at  mr  at  lu  tgmitim.  TMt 
■Mt  aHjr  Mt  b«  nlMiad  toropw  paMealtM  wul 
I  feM*  damd  kjf  dw  ifyfnfriata  mSutaiy  mniet  at 


STUDY 

PROJECT 


CONGRESS  AND  ARMY  OPERATIONAL 
TEST  AND  EVALUATION  \ 


MR.  FRANK  JOHN  APICELLA,  DAC 


DISTKZIUTtON  STATEMENT  A:  ANprefVc4  for  puUle 
relcsMf  distribution  Is  ualtsdtsd. 


10  APRIL  1989 


DTIC 

ELECTE  M 

jUL0  51989|  ■ 

HP  E 


U.S.  ARMY  WAR  COliEGE,  CARUSU  BARRACKS,  PA  17013-5050 


89  7  03  008 


Un^assif  ied 


:ECU~ITV  classification  of  this  °aGE  'MTion  Dnia  Hnrrrrdi 


DO  1  JAM  "Vs  1473  eomon  OF  »  NOV  6S  is  OBSOLETE 


Unclassified _ 

OEC’JRITY  cl  A5SI  F'C  a  rrov  Tm»S  PA'»e  D»IS  Enterod) 


Unclassified _ 

SECURITY  CLASSIFICATION  Oi  THIS  P HOEfirhan  Omf  ent9r»€f) _ 

20.  ABSTRACT — continued. 

j  roles  of  the  Defense  Operational  Test  and  Evaluation  Office  (DOT&E)  are 

I  discussed.  The  problems  and  impacts  of  new  Survivability/Vulnerability 

"live-fire"  legislation  are  explored.  Further,  the  primary  and  second  order 
effects  associated  with  the  strict  adherence  to  the  recent  congressional 
mandate  which  precludes  any  participation  of  weapon  system  contractors  or 
subcontractors  in  the  operational  testing  and  evaluation  of  their  system 
are  also  assessed.  An  overall  evaluation  of  the  effectiveness  of  Army 
Operational  Test  and  Evaluation  and  20  recommendations  for  improvement 
are  offered  for  consideration. 


Unclassified 


security  classification  of  This  ^AGerrtTisn  Data  Enlarad.t 


USAWC  MILITARY  STUDIES  PROGRAM  PAPER 


The  views  expressed  Is  this  psper  ere  thoss  of  the 
author  end  do  sot  seceassrlly  reflect  the  views  of 
the  Depsrtaent  of  Defense  or  any  of  its  sfencies. 

This  doC'  lent  «sy  not  he  released  for  open  publication 
until  It  hss  been  cleared  by  the  appropriate  nllitsrv 
service  or  govemsient  agency. 


CONGRESS  AND  ARMY  OPERATIONAL  TEST  AND  EVALUATION 

AN  INDIVIDUAL  STUDY  PROJECT 


by 


Mr.  Frank  John  Apicella,  DAC 


Colonel  Charles  S.  Palmer 
Project  Adviser 


DlSTniuncV  STATBKEST  i:  Appeoved  for  puUic 
relssssf  dlstrlbutioti  Is  unlisted* 

U.S.  Army  War  College 
Carlisle  Barracks,  Pennsylvania  17013 
10  Apr i 1  1989 


ABSTRACT 


AUTHOR:  Mr.  Frank  John  Apicella,  GM-15,  DAC 

TITLE:  Congress  and  Army  Operational  Test  and  Evaluation 

FORMAT:  Individual  Study  Project 

DATE:  10  April  1989  PAGES:  50  CLASSIFICATION:  Unclassified 

-i 

Congress  has  become  increasingly  more  interested  in  the 
servi  ces' materi  el  acquisition  processes.  A  primary  -focus  of 
this  new  concern  has  been  major  weapon  system  operational  test 
and  evaluation.  This  study  examines  the  merits  and  demerits  of 
congressi onal  initiatives  to  improve  the  overall  quality  of 
operational  test  and  evaluation,  as  they  relate  to  the  United 
States  Army.  ’  The  rationale  for  the  establishment,  the  current 
effectiveness,  and  possible  future  roles  of  the  Defense 
Operational  Test  and  Evaluation  Office  (D0TS<E)  are  discussed. 
-The  problems  and  impacts  of  new  Survi vabi 1 i ty/Vul nerabi 1 i ty 
"live-fire"  legislation  are  explored.  Further,  the  primary  and 
second  order  effects  associated  with  the  strict  adhsrvncs  to 
the  recent  congressional  mandate  which  precludes  any 
part i ci parti  on  of  weapon  system  contractors  or  subcontractors 
in  the  operational  testing  and  evaluation  of  their  system  are 
also  assessed.',  An  overall  evaluation  of  the  effectiveness  of 
Army  Operational  Test  and  Evaluation  and  20  r ecommendat i onE  for 
improvement  are  offered  for  consideration. 


i  i 


TABLE  OF  CONTENTS 


Page 


ABSTRACT .  ii 

LIST  OF  ILLUSTRATIONS .  iv 

CHAPTER  I.  INTRODUCTION .  1 

General . 1 

De-finitions .  4 

II.  OPERATIONAL  TEST  AND  EVALUATION . 

Origins . 

Army  OT?<E .  1 

III.  DEFENSE  OPERATIONAL  TEST  AND  EVALUATION 

OFFICE .  17 

IV.  SURVIVABILITY/VULNERABILITY  LEGISLATION- 

"WHISTLE-BLOWERS" .  24 

V.  CONTRACTOR  PARTICIPATION  LEGISLATION- 

GOVERNMENT  AUDIT  ORGANIZATION .  30 

VI.  CONCLUSIONS . 37 

VII.  RECOMMENDAT I ONS .  39 

ENDNOTES .  45 

BIBLIOGRAPHY .  46 


O'  O'  t  o 


LIST  OF  ILLUSTRATIONS 


EXHIBIT  1 


ACQUISITION  PROGRAMS . 

ORDER  OF  PRECEDENCE  -  ACQUISITION  STRATEGY 

ARMY  MATERIEL  ACQUISITION  PROGRAM 
CATEGORIES  AND  DECISION  AUTHORITY . 

TEMP  STAFFING  PROCESS . 

T2<E  OPERATIONAL  RELATIONSHIP . 


Pc(y  e 


h 


.  7 

20 


i  V 


CONGRESS  AND  OPERATIONAL  TEST  AND  EVALUATION 


CHAPTER  I 

INTRODUCTION 

GENERAL 

On  9  January  1989  in  his  first  message  to  Congress  our 
newly  selected  Commander-i  n-Chi  e-f  encouraged  them  to  refrain 
from  mi cro-managi ng  the  armed  services  procurement  processes. 

At  that  very  time  a  distinguished  ex-member  of  Congress  and  the 
Senate  Armed  Services  Committee  was  being  persecuted  by  his 
peers  as  part  of  the  Secretary  of  Defense  confirmation  process 
for  allegedly  accepting  defense  contractor  payments  of  *750, 000 
in  honoraria  and  consultant  fees.  Members  of  the  military 
services  tend  to  be  critical  of  outside  Congressional  influence 
on  their  day-to-day  management  attivities;  especially  when 
these  Congressional  actions  are  rationalized  by  purported 
wrongdoing,  allegations  of  incompetence,  and  media  theatrics. 
Finding  fault  in  any  large  organization  is  not  a  difficult 
task.  In  the  last  30  years  Congressional  staffs  have  increased 
in  number  by  an  order  of  magnitude  from  6,400  in  1960  to  17,137 


in  1980  to  over  19,000  in  1989. 


These  sta-f-fs  -frequently  assume 


fault  finding  missions  in  search  of  new  areas  where 
Congressi onal  influence  may  be  applied.  Over  55,000  written 
Congressional  inquiries  were  addressed  to  the  Secretary  of  the 
Army  in  1988  alone.  Military  procurement  has  been  a  popular 
hunting  ground.  Critics  of  this  new,  substantial  Congressional 
interest,  are  concerned  about  the  level  of  micro-management, 
fairness,  bias,  and  ethical  implications  associated  with  these 
Congressional  "witch  hunts".  Surprisingly,  select  members  of 
Congress  and  the  military  establishment  are  finding  themselves 
and  their  actions  visible  and  subject  to  reproach.  The 
question  is:  are  the  best  interests  of  the  American  fighting 

man  being  served? 


The  materiel  acquisition  process  has  been  an  area  of 
considerable  Congressional  interest  for  many  years.  Congress 
controls  the  spending  of  a  staggering  amount  of  money  each 
year . 


"Annual  purchases  by  DOD  total  almost  $170 
billion  more  than  the  combined  purchases  of  General 
Motors,  E>!>:on,  and  IBM.  The  DOD  research  and 
development  expend! tures ’are  more  than  fifteen 
times  those  of  France,  Germany,  or  the  United 
Kingdom,  and  eighty  times  those  of  Japan.  Defense 
acquisition  involves  almost  15  million  separate 
contract  actions  per  year  -  or  an  average  of  56,000 
contract  actions  every  working  day."  1./ 

The  efficient  and  responsible  spending  of  funds  is  vital  to 

ensure  the  safety  of  the  nation  and  the  effectiveness  of  our 

Armed  Services.  These  funds  also  have  significant  secondary 


implications  with  respect  to  defense  contracts  and  employment 


opportunities  in  many  o-f  the  states.  Contractors  -frequently 
subcontract  parts  ot  large  procurement  projects  to  companies 
located  in  the  state  of  influential  Congressmen.  The  dangers 
here  are  that  these  interests  can  potentially  influence 
congressmen  to  champion  specific  programs  being  produced  in  his 
or  her  district  or  potentially  bias  the  military  in  their 
procurement  and  fielding  recommendations. 

Fortunately,  Congress  has,  by  and  large,  acted  responsibly 
and  the  American  fighting  man  has  been  provided  with  weapon 
systems  and  support  equipment  that  are  the  finest  in  the 
world.  However,  it  is  important  to  note  that  while  these 
systems  are  unquestionably  the  best,  they  are  not  perfect. 
Critics  very  rarely  attack  the  comparative  performance  of  new 
developmental  versus  current  systems  or  other  nations'  weapon 
systems.  However,  finding  fault  is  not  difficult.  One  area 
that  has  been  particularly  lucrative  is  biased  appraisals  of 
the  operational  test  and  evaluation  of  developmental  systems. 
The  focus  of  this  study  will  be  on  the  Army's  operational  test 
and  evaluation  (□TS<E)  process  as  opposed  to  the  entire  materiel 
acquisition  process  <MAP) .  However,  the  lessons  associated 
with  Congressional  involvement  in  OT?/:E  are  just  as  applicable 
to  the  MAP.  Further,  several  of  the  lessons  learned  could  be 
applied  to  many  of  the  other  new  frontiers  that  Congress  is 


currently  exploring. 


DEF  I  hi  I T  I  DNS 


The  primary  purpose  -for  conducting  test  and  evaluation  was 
of-fered  in  a  1986  report  to  the  Secretary  o-f  De-fense: 

"...it  IS  essential  that  the  goal  o-f  the 
acquisition  process  never  be  lost  sight  o-f: 
providing  the  users — the  young  men  and  women  in  the 
field — weapons  and  equipment  that  work  as  the  user 
needs  them  to  and  are  available  when  and  where  they 
are  needed.  This  will  be  assured  only  if  decision 
makers  have  the  benefit  of  the  unfiltered  views  and 
findings  of  an  independent  DTJ-rE  community  based  upon 
the  best  possible  ’Will  it  work?’  -oriented  test  and 
evaluation  of  the  operational  effectiveness  and 
suitability  of  the  system  in  question,  al wavs 
including  realistic  field  testing  of 

pr oduc 1 1  on-r epr esentat  1  ve  test  articles.  These  OT?<E 
results  must  be  candidly  and  responsibly  reported, 
providing  the  best  judgement  on  whether  the  system 
will  (or  early  in  the  program,  is  likely  to)  meet 
the  operational  requirements  of  the  user."  2/ 


A  basic  understanding  of  the  terminology  associated  with  the 
riAP  and  OT!?<E  is  imperative  in  comprehending  these  areas. 

First,  one  must  be  sensitive  to  the  fact  that  we  are  not  just 
talking  solely  about  the  item,  the  weapon.  System  or  materiel 
acquisition  includes  many  other  critical  elements;  operator 
and  maintainer  training  programs,  associated  support  items, 
detailed  manuals,  spare  parts,  tactics  and  doctrine  (which 
integrates  and  exploits  the  full  technical  capabilities  of  the 
new  system),  and  a  host  of  other  critical  considerations  which 
are  absolutely  essential  to  support  the  effective  fielding  of  a 
major  weapon  system.  Secondly,  there  are  two  major  factors  that 
govern  the  nature  of  the  acquisition  process  of  these  complete 


systems : 


the  acquisition  strategy  selected  and  the  level  of 


DOD/DA  management  exercised.  With  regard  to  acquisition 
strategy,  the  worst  case  and  typically  most  costly  acquisition 
is  a  new  development  program  to  produce  an  entirely  new  weapon 


system.  (See  EXHIBIT  1). 

EXHIBIT  1 

ACQUISITION  PROGRAMS* 


THE  MATERIEL  ACQUISITION  PROCESS 

TRADITIONAL  -  A  PERIOD  OF  APPROXIMATELY  TWELVE  (12)  YEARS  REQUIRED  TO 
DEVELOP  AND  ACQUIRE  A  NEW  SYSTEM  WHICH  INCLUDES  THE  DESIGN,  FABRICATION, 

AND  DEVELOPMENT  OF  THE  ITEM  OF  EQUIPMENT,  TECHNICAL  MANUALS,  THE  TACTICS 
AND  DOCTINE  FOR  ITS  EMPLOYMENT,  TRAINING  PROGRAMS  FOR  OPERATORS  AND 
MAINTAINERS,  A  COMPREHENSIVE  LOGISTIC  SUPPORT  STRUCTURE,  AND  A  MERIAD  OF 
OTHER  REQUIREMENTS  NECESSARY  TO  FULLY  SUPPORT  THE  EFFECTIVE  FIELDING  OF  A 
MAJOR  WEAPON  SYSTEM  ARMY  WIDE  THROUGHOUT  THE  WORLD.  THIS  PROCESS  CONSISTS 
OF  FOUR  PHASES;  "ONCEPT  EXPLORATION,  DEMONSTRATION  AND  VALIDATION,  FULL 
SCALE  DEVELOPMENT,  AND  PRODUCTION  AND  DEPLOYMENT. 

STREAMLINED  -  AN  EXPEDITED  VERSION  OF  THE  TRADITIONAL  PROCESS  REQUIRING 
APPROXIMATELY  EIGHT  (S)  YEARS.  THIS  PROCESS  CONSISTS  OF  THREE  PHASES; 

PROOF  OF  PRINCIPLE,  DEVELOPMENT  AND  PRODUCTION  PROVE  OUT,  AND  PRODUCTION 
AND  DEPLOYMENT. 

CATEGORIES  OF  NQNDEVELOPMENTAL  ITEM  (NDI)  ACQUISITION 

CATEGORY  A  -  AN  OFF-THE-SHELF  ITEM  THAT  CAN  BE  PROCURED  COMMERCIALLY,  FROM 
ANOTHER  SERVICE,  GOVERNMENT  AGENCY,  OR  A  FOREIGN  COUNTRY  THAT  REQUIRES  NO 
ENGINEERING  DEVELOPMENT  TO  MEET  THE  ARMY’S  REQUIREMENTS.  MANY  OF  THE 
PREFIELDING  REQUIREMENTS,  SUCH  AS  DEVELOPING  TRAINING  PROGRAMS,  SPARE 
PARTS  AND  MAINTENANCE  PLANS,  AND  OTHERS  STILL  HAVE  TO  BE  ACCOMPLISHED; 

THUS,  THE  TIME  REQUIRED  FOR  ACQUISITION  MAY  BE  TWO  (2)  YEARS  (OR  LONGER). 

CATEGORY  B  -  AN  ITEM  THAT  DOES  REQUIRE  SOME  RUGGEDIZING  AND  MILITARIZING 
PRIOR  TO  BEING  ACCEPTABLE  FOR  ARMY  USE.  THEBE  PROGRAMS  REQUIRE  SOME 
ENGINEERING  DEVELOPMENT  AND  VERIFICATION  TESTING  AND  MAY  REQUIRE  THREE  AND 
A  HALF  TO  FOUR  (4)  YEARS  FOR  ACQUISITION. 

CATEGORY  C  -  AN  ITEM  THAT  CONSISTS  OF  SEVERAL  CURRENTLY  FIELDED  AND/OR 
COMMERCIALLY  AVAILABLE  ITEMS  WHICH  HAVE  BEEN  INTEGRATED  AND  THAT  ONLY 
REQUIRES  A  LIMITED  AMOUNT  OF  DEVELOPMENT  BEFORE  IT  IS  ACCEPTABLE  FOR  ARMY 
USE.  INTEGRATION  TIME,  ENGINEERING  DEVELOPMENT,  TECHNICAL  AND  OPERATIONAL 
TESTING,  AND  OTHER  PREFIELDING  REQUIREMENTS  MAY  RESULT  IN  THESE  PROGRAMS 
BEING  COMPLETED  IN  FIVE  (5)  YESRS. 

PRODUCT  IMPROVEMENT 

PRODUCT  IMPROVEMENT  PROGRAM  (PIP)  -  IS  A  REFERBISHMENT  OF  AN  EXISTING  ITEM 
OF  ARMY  EQUIPMENT  TO  EXTEND  ITS  USEFUL  LIFE,  PRECLUDE  THE  NEED  FOR  A  NEW, 
COSTLY  REPLACEMENT  ACQUISITION  PROGRAM,  AND  TO  IMPROVE  THE  EXISTING 
CAPABILITIES  OF  THE  SYSTEM. 

PREPLANNED  PRODUCT  IMPROVEMENT  (P3I)  -  IS  A  PREPROGRAMMED  SCHEDULE  OF 
IMPROVEMENTS  TO  A  SYSTEM  TO  EXPLOIT  EMERGING  TECHNOLOGIES  AND  TO  REDUCE 
THE  TIME  AND  RISK  ASSOCIATED  WITH  FIELDING  THE  INITIAL  SYSTEM. 


»  ACQUISITION  TIMES  INDICATED  ABOVE  ARE  ESTIMATES  ONLY.  THE  TIME  REQUIRED 
FOR  THE  DEVELOPMENT  AND  ACQUISITION  OF  EACH  INDIVIDUAL  SYSTEM  VARIES. 


Another  case  is  the  o-f -f -the-shel  f  or  Non-devel  opmental  Item 
(NDI)  which  involves  the  procurement  and  adoption  n-f  an  item 
available  in  the  private  sector  -for  military  use  (there  are 
three  categories  o-f  NDI  programs).  The  third  program  type  is  a 
materiel  or  product  improvement  o-f  an  existing  military 
system-  This  last  approach  is  a  preferred  strategy  or  program 
since  it  is  typically  the  least  costly  (at  least  over  the  short 
term),  requires  less  time  than  full  development,  and  has  less 
technical  risk  associated  with  it.  (See  EXHIBIT  2) 


EXHIBIT  2 


ORDER  OF  PRECEDENCE  -  ACQUISITION 


MOTE:  PME-PUMMEO  PROOOCT  IMPROVEHINT  (P3I)  IS  USED  IN  CONJUNCTION  WITH  THESE 
APPROACHES  FOR  MAXIMUM  lENEFIT  AT  LEAST  COST,  TIME,  AND  RISK 

Vference:  AR  70-1 
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The  documentation,  test,  and  funding  requirements  are  highly 
dependent  upon  how  an  acquisition  program  is  labeled. 
Variations,  permutations,  and  clever  labeling  and  irrational 
accommodations  of  these  different  types  of  acquisition  programs 
are  not  uncommon.  There  are  five  system  acquisition 
categories:  DOD  major.  Army  designated  major  programs  (DAP), 

and  In-Process  Review  (IPR)  systems  Categories  1  through  3. 

(See  EXHIBIT  3) . 


EXHIBIT  3 


M^utoMon  pfOQfMi 


n«gram  lifpc  and  category 


MOAP 

OABIeral 

Component  (Army)  level 


Nonmeior  level  I 


Nonmeior  level  i 


Nonmeior  level  I 


Program  managamara 


PEO/PM 


P60/PM 


P60/PM 


PEO/PM 


Protect  offioorv  or  ooutvitenl 
(deaigneled  by  MATDEV) 


Syetwne  meneger.  commodity 
meneoer,  or  equnmient  (eeaigned 
by  MATDEV/POE  oontar) 


MteatotM  review  i 


ASARC 


ASARC 


Progrant  deciaion  auetoithr 


SECOEF 
(DAE  Agent) 
SECARMY 
(ME  Agent) 


MATDEV  Commander 


MATDEV  Commendar 


Al  taveli  are  gavamed  by  tie  prinolplae  of  AM  70-1:  however,  tw  MATDEV  may  MIor  tie  dedplined  management  review  iorxane  tor  tmrato  R  and  ■ 


DOD  major  and  DAP  systems  char  acter  i  st  i  cal  1  y  require  e;;  tensive 
engineering  development  and  testing.  There  are  several  o-f  these 
systems  which  are  o-f  general  Congressional  interest  because  of 
their  high  cost  and  importance.  IPR  systems  typically  require 
less  engineering  development  and  testing  effort  and  are 
occasionally  subject  to  selective  Congressional  interests. 
Lastly,  there  are  three  types  of  testing  that  are  part  of  the 
MAP.  Force  Development  Testing  (FDT)  is  conducted  to  assist  in 
refining  training  programs  and  in  the  development  of  tactics 
and  doctrine  for  new  systems.  Technical  testing  (TT>  is  a 
developmental  or  confirmatory  process  whereby  the  item  or  a 
subcomponent  is  tested  in  a  controlled  environment  to  ascertain 
or  to  verify  whether  the  current  design  meets  or  exceeds  system 
performance  requirements.  Operational  testing  (OT)  is  a 
confirmatory  process  whereby  the  system  (to  include  the 
soldier,  associated  support  items  of  equipment  (ASIDE),  the 
training  program,  selected  levels  of  maintenance,  and  as  much 
of  the  total  system  as  possible)  is  tested  in  an  operational 
(simulated  wartime)  environment  to  verify  whether  the  final 
production  system  i s  or  has  the  potential  to  be  operationally 
effective  and  suitable  for  Army  use.  DODI  5000.2  attempts  to 
highlight  just  some  of  the  elements  of  this  operational 
responsibility  in  defining: 

"...Operational  Effectiveness.  The  overall  degree 
of  mission  accomplishment  of  a  system  when  used  by 
representative  personnel  in  the  environment  planned 
or  expected  for  operational  employment  of  the  system 
considering  organization,  doctrine,  tactics, 
survivability,  vulnerability,  and  threat  (including 
countermeasures,  nuclear,  and  chemical  and/or 
biological  threats)... 
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. ■ ■ Operat i onal  Suitability.  The  degree  to  which  a 
system  can  be  placed  satisfactorily  in  field  use 
with  consideration  given  to  availability, 
compatibility,  transportability,  interoperability, 
reliability,  wartime  usage  rates,  maintainability, 
safety,  human  factors,  manpower  suppor t ab i 1 i ty , 
logistics  supportabi 1 i ty,  documentation,  and 
training  requirements."  3/ 


The  key  difference  between  TT  and  OT  is  that  technical  testing 
is  typically  an  iterative  process  of  controlled  "test-f  i  >;-test " 
where  operational  testing  is  a  confirmatory  test  of  the  entire 
system  conducted  in  an  operational  environment  to  determine 
overall  system  suitability  for  Army  use.  FDT  are  not  always 
conducted;  however,  more  interest  has  been  placed  on  trying  to 
conduct  these  valuable  tests  prior  to  the  start  of  OT.  It  has 
been  very  difficult  conducting  OT's  without  well  thought  out 
tactics  and  doctrine.  Combat  developer  approved  tactics,  which 
exploit  new  system  capabilities,  have  been  particularly 
difficult  to  acquire  prior  to  start  of  OT. 


CHAPTER  II 

OPERATIONAL  TEST  AND  EVALUATION 

ORIGINS 

Army  test  and  evaluation  probably  had  its  origins  when 
some  of  the  first  pieces  of  equipment  were  being  considered  for 
purchase  by  the  government  for  military  use.  An  early  example 
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is  in  1820  when  John  Hall  o-f-fered  prototype  breach-loading 
ri-fles  to  the  Army  -for  test.  The  weapons  were  tested  and 
deemed  unsati  s-f  actory  tor  use  in  combat  and  were  not  procured. 
The  inability  ot  the  Army  to  efficiently  develop,  test,  and 
acquire  sutticient  quantities  (i.e.,  standard  issue  was  key  to 
resupply)  ot  the  best  weapon  systems  had  dire  consequences. 
During  the  Civil  War  soldiers  tought  with  substandard  weapons, 
to  include  obsolete  muzzle-loaded  muskets.  In  1919  the  Army 
tested  motor  trucks  to  replace  the  tield  wagon. 

"The  quar termaster  department  at  that  time 
began  a  developmental  and  testing  program  to  produce 
a  suitable  truck.  A  test  o-f  two  Army  trucks  and  a 
privately-owned  truck  which  was  run  from  Washington, 

DC  to  Ft  Benjamin  Harrison  covered  1,524  miles  in 
forty-eight  days.  The  conclusion  was  reached  that 
although  the  results  were  not  entirely  satisfactory, 
trucks  did  show  some  promise."  4/ 

Other  tests  included  those  of  Col-  Billy  Mitchell  who  in  1921 
demonstrated  the  effectiveness  of  Army  air  against  shipping. 

In  1924  the  concept  of  conducting  separate  engineering  and 
"service"  (operational)  tests  was  formalized  in  AR  850-25, 
Types  of  Equipment  Used  by  the  U.S.  Army.  However,  War 
Department  (DOD) ,  industry,  and  -Congr essi onal  cooperation  in 
materiel  acquisition  did  not  evolve  appreciably  until  WW  II. 
Along  with  the  urgencies  involved  in  America’s  requirement  to 
mobilize  quickly,  we  became  sensitized  to  the  importance  of 
timely  production  and  technological  advantages.  Early  in  the 
war,  German  Tiger  Tanks  were  so  superior  that  American  weapon 
systems  were  unable  to  stop  them. 
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The  relationship  between  industry.  Congress,  and  DOD 


reached  its  height  during  WW  II  when  the  nation’s  industrial 
base  was  thoroughly  committed  to  mobilization  and  technological 
development.  DOD  and  industry  worked  in  partnership  to  produce 
"the  best  and  the  most"  in  support  o-f  the  American  -fighting  man 
and  the  alliance.  Since  this  last  declared  war,  the 
relationship  between  industry,  DOD,  and  Congress  has  su-f-fered 
and  become  more  adversarial.  Today,  weapons  are  -far  more 
complex,  costly,  and  e-f-ficient.  The  numbers  o-f  items  procured 
has  become  limited,  competition  has  become  sti-f-fer,  and  design 
risks  have  become  -far  greater.  Collectively  all  these  -factors 
have  created  an  environment  which  is  more  structured,  strict, 
and  accountable.  During  the  50'' s  and  60’ s,  especially  with  the 
paranoia  associated  with  the  launch  o-f  Sputnik  in  1958,  a 
considerable  amount  o-f  emphasis  was  placed  on  developing  and 
acquiring  high  technology  weapon  systems.  Also,  much  emphasis 
was  placed  on  engineering  testing,  and  to  a  lesser  degree 
"service"  testing.  In  the  late  60’s,  in  part  due  to  poor 
per-formance  noted  in  several  o-f  our  weapon  systems  in  Vietnam, 
the  services  became  concerned  over  undue  emphasis  on 
engineering  testing  as  opposed  to  "service"  (operational) 
test i ng . 


"The  war  in  Vietnam  raised  numerous  questions 
regarding  □TS<E....  In  February  1970,  the  Air  Force 
Scientific  Advisory  Board  published  a  report 
regarding  the  differences  between  operational  test 
results  and  the  results  of  actual  i n-the-f i el d , 
hands-on  experience.  The  Scientific  Advisory  Board 
concluded  that  ’false  confidence  was  held  in  certain 
test  results  which  was  due  to  insufficient  data 
accumulated  during  Category  I  and  II  test 
programs’ — testing  conducted  by  the  contractor  and 
developer,  with  little  input  from  the  user."  5/ 
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Gn  1  July  1970  "The  Report  to  the  President  and  the 
Secretary  o-f  De-fense  on  the  Department  o-f  De-fense"  by  the 
Blue  Ribbon  De-fense  Panel  stated; 

"4.  The  quality  o-f  QT8<E  is  very  uneven.  There  is 
no  question  that  military  OTStE  can  and  should  be 
planned  and  executed  much  more  e-f -f  ecti  vel  y  than  it 
has  in  recent  years. 

5.  The  results  o-f  QTS<E  which  has  been  accomplished 
have  not  been  adequately  made  available  or  used  by 
DOD  agencies  which  need  them.  There  is  no  method  o-f 
evaluating  and  preserving  such  information 

6.  0T8<E  is  not  adequately  managed  or  supervised  at 
the  OSD  level."  6/ 

In  1971  Deputy  Secretary  o-f  De-fense  David  Packard  directed  the 
service  Chie-fs  to  establish  independent  operational  test  and 
evaluation  organisations.  As  a  result  each  service  created  an 
0TS<E  organisation  in  the  early  70’s.  (Congress  did  not 
legislate  the  creation  o-f  a  DOD  level  0T8<E  o-f-fice  until  1983.) 
Department  of  the  Army  formed  U.S.  Army  Operational  Test  and 
Evaluation  Agency  (OTEA)  at  Ft.  Belvoir,  Va.  on  25  September 
1972  (Today  located  at  The  Park  Center  II  Building  in 
Alexandria,  Va. ) .  The  original  mission  of  OTEA  was; 


"...Accomplish  the  planning  for,  direction  of,  and 
evaluation  of  operational*  testing  of  all  major 
systems  and  selected  non-major  systems  required  in 
the  materiel  acquisition  process. 

...Coordinate  Force  Development  Testing  and 
Ex  per i men  t  a t i on . 

...Coordinate  Army  participation  in  the  planning  and 
conduct  of  joint  and  combined  operational  test 
activities. 

...Provide  a  strong  focal  point  organisation  at 
Headquarters  DA  to  keep  DA  and  OSD  fully  informed  on 
the  Army’s  operational  test  and  evaluation  needs  and 
accomplishments."  7/ 


The  other  services  also  complied,  the  Navy  with  Operational 
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Test  and  Evaluation  Force  (OPTEVFOR) ,  the  Air  Force  Operational 
Test  and  Evaluation  Command  (AFOTEC) ,  and  the  Marine  Corps 
Operational  Test  and  Evaluation  Agency  (MCOTEA) . 


ARMY  OTg<E 

OTEA  is  a  field  operating  agency  of  the  Office,  Chief  of 
Staff,  Army  (0C3A) ,  Its  current  mission  is  to  support  the  MAP 
by  managing  the  Army^’s  continuous,  comprehensive  evaluation 
(C2E)  and  user  test  programs.  The  original  organizational 
concept  for  OTEA  was  well  thought  out,  effective,  and 
efficient.  It  would  have  even  made  former  Secretary  of 
Defense,  Robert  S.  McNamara,  proud.  Two  of  the  primary 
considerations  in  forming  OTEA  were  independence  of  (1) 
operational  testing  and  (2)  independence  of  operational 
evaluation.  A  key  conclusion  of  the  1970  Blue  Ribbon  Defense 
Panel  was  that  "...it  was  not  in  the  interest  of  unbiased  and 
objective  operational  test  and  evaluation  to  have  those  who 
perform  it  report  through  the  developer..."  One  of  the 
perceived,  critical  deficiencies  of  previous  test  and 
evaluation  processes  was  the  lack  of  independence.  Even  though 
problems  were  discovered,  lack,  of  independence,  at  times, 
resulted  in  rationalization  which  minimized  the  importance  of 
test  findings.  This  significantly  impaired  a  process  which 
should  have  otherwise  communicated  these  problems  to  the 
highest  level  of  decision  making.  As  a  result  OTEA  is 
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commanded  by  a  Major  General  who  works  directly  -for  the  Chiet 
o-f  Sta-f-f,  Army  (CSA)  and  QTEA  has  a  Senior  Executive  Service 
<SES)  Scientific  Advisor  (or  Technical  Director)  who  maintains 
a  close  working  relationship  with  the  Army  Secretariat.  This 
balanced  relationship  between  the  OTEA  commander-and-the  CSA 
and  the  OTEA  Scientific  Advi sor-and-the  DOD/Army  Secretariat 
has  been  a  major  factor  in  contributing  to  the  success  and 
independence  of  the  Army’s  OT?<E  efforts. 

□TEA  is  a  very  small  organization  composed  of  less  than 
250  professional  people,  approximately  half  of  them  are 
military  and  half  are  civilian.  OTEA  is  charged  with  the 
responsibility  for  the  programming,  design,  conduct  and 
evaluation  of  operational  testing  of  major  (and  selected 
non-major)  Army  materiel  acquisitions.  Also,  C2E 
responsibility  for  IPR  Category  (level)  2  and,  to  some  extent, 
Category  (level)  3  systems  is  a  newly  assumed  mission.  OTEA’s 
required  level  of  effort  in  order  to  perform  this  new  mission 
has  been  difficult  to  estimate.  Performing  this  new  mission 
will  entail  a  considerable,  total  acquisition  process 
evaluation  responsibility  for  several  hundred  weapon  systems  or 
combat  support  and  combat  service  support  equipment.  The  exact 
number  of  these  programs  has  been  very  difficult  to  determine 
since  individual  programs  are  managed  with  varying  levels  of 
intensity  by  TRADOC  schools.  Health  Services  Command, 
Informations  Systems  Command  and  others.  The  Army  leadership’s 
focus  is  only  on  several  of  the  higher  priority  programs. 
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There  is  no  central  management  o-f  these  several  hundred  lesser 
Army  materiel  acquisition  programs;  even  though,  it  has  been 
argued  that,  collectively  these  lesser  systems  may  represent  a 
considerable  portion  o-f  the  defense  budget.  DTEA  is  in  the 
early  stages  of  compiling  an  accurate  list  of  active  materiel 
acquisition  programs;  funding/cost  information  on  all  these 
programs  is  not  available- 

□TEA's  preoccupation  with  strengthening  and  expanding 
the  scope  of  its  evaluation  mission  has  had  some  negative 
effects.  QTEA’s  independent  operational  testing  mission  has 
significantly  diminished  to  a  point  where  the  agency  will 
conduct  or  closely  manage  only  one  or  two  operational  tests  per 
year.  Originally,  OTEA  was  charged  with  performing  both  (1) 
the  independent  operational  testing  and  (2)  the  independent 
operational  evaluation  mission  for  Army  major  and  selected 
non-major  weapon  systems.  Today,  OTEA  has  significantly 
expanded  the  scope  of  its  independent  evaluation  mission  (C2E) 
to  include  total  life  cycle  evaluation  of  many  more  systems  and 
has  delegated  nearly  all  of  its  independent  operational  testing 
mission  to  Training  and  Doctrine  Command’s  (TRADOC)  test 
boards.  TRADOC  is  typically  the  combat  developer  for  the 
system  being  tested.  The  merits  of  these  actions  are  many; 
however,  the  chief  advantage  is  that  the  test  boards  are  better 
suited  for  the  testing  of  complex  weapon  systems.  Many  of 
these  systems  require  the  use  of  sophisticated  instrumentation 
and  facilities  to  produce  accurate,  meaningful  test  results. 
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Further,  OTEA's  evaluation  mission  has  pro-fited  -from  having  its 
evaluators  being  privy  and  a  party  to  the  whole  development 
process.  Evaluators  participate  in  both  technical  and 
operational  planning  et-forts  as  a  member  o-f  individual  weapon 
system  acquisition  teams.  This  has  significantly  increased  the 
evaluator’s  understanding  o-f  the  system,  its  status,  and  how 
■far  it  has  progressed  in  development.  The  con  o-f  this  approach 
is  that  the  operational  testing  process  has  lost  a  considerable 
amount  o-f  independence  and  one  might  argue  that  more  o-f  the 
evaluation  -focus  is  being  placed  upon  engineering  assessment  as 
opposed  to  operational  considerations.  True  independence  will 
be  di-f-ficult  to  attain  until  both  the  operational  testing  and 
operational  evaluation  missions  for  all  new  weapon  systems  are 
consolidated  into  a  separate,  Army  independent  operational  test 
and  evaluation  command.  Authority,  leadership,  command  lines, 
test  facilities,  and  staffing  will  be  key  to  the  success  of 
such  a  command.  OTS<E  must  be  uncompromising  and  responsibility 
for  its  conduct  must  be  kept  at  the  service  level.  OTS<E  focus 
must  remain  on  selecting  the  best  materiel  for  the  soldier, 
sailor,  or  aviator.  In  this  age  of  limited  resources,  higher 
level  authority  must  have  the  benefit  of  unbiased  OT?tE  results 
to  facilitate  the  decision  making  processes  which  must 
prioritize  and  make  compromises  for  the  overall  good  of  the 
service  or  DOD. 
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CHAPTER  III 


DEFENSE  OPERATIONAL  TEST  AND  EVALUATION  (DQTg<E)  OFFICE 

Congressional  concern  regarding  the  adequacy  of  DDD 
management  of  □T?<E  goes  back  to  the  1970  Blue  Ribbon  Defense 
Panel  :  "QT?<E  is  not  adequately  managed  or  supervised  at  the 

OSD  1 evel ■ "  As  a  result  of  increased  media  and  Congressional 
interest  in  the  MAP,  legislation  was  introduced  by  Senators 
David  Pryor  (Democrat  from  Arkansas)  and  Senator  William  Roth 
(Republican  from  Delaware)  for  the  creation  of  a  DDT8<E  office 
in  April  1983.  Congress’s  rationale  for  establishing  this 
office,  Public  Law  98-94,  is  perhaps  best  captured  in  a  letter 
from  Congress  to  Secretary  of  Defense  Carlucci  on  9  December 
1987.  The  purpose  of  this  letter  was  to  inform  the  Secretary 
of  Congressional  concern  over  a  significant  loss  of 
i  ndependence  which  could  result  from  DDT8<E’5  plans  to  broaden 
its  mission  to  include  developmental  T?<E. 


"In  1983,  the  Congress  voted  overwhelmingly 
in  favor  of  creating  an  Independent  Director  of 
Operational  Testing  and  Evaluation.  We  did  this  to 
make  certain  that  the  office  charged  with  analyzing 
the  vital  field  tests  of  our  weapons  would  not  in 
any  way  be  compromised  by-  ties  with  the  weapons 
development  community,  including  the  services  and 
contractors,  who  created  and  advocated  the  weapon 
systems.  Before  Congress  took  action,  as  Senator 
Roth  stated  on  the  floor  of  the  Senate  last  year: 

’operational  testing  was  controlled  by  the 
same  man  who  controlled  research,  development  and 
acquisition.  And  we  had  problems.  We  had  cheating 
in  tests;  we  had  rigged  tests;  sometimes  we  had  no 
tests;  and  often  we  had  self-serving  test  analysis 
and  reports  influenced  heavily  by  those  who  did  not 
want  to  see  their  own  handiwork  objectively 
assessed . ’ "  8/ 
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DOT?-:E  was  established  in  Nov  1983  under  the  temporary 
leadership  o-f  Dr.  Isham  W.  Linder,  Director  of  Defense  Test  and 
Evaluation.  Another  layer  of  review  was  imposed  upon  an 
already  review  laden  MAP. 

DOTScE  enjoys  a  rather  unique  position  in  the  MAP  in  that 
it  approves  system  Test  and  Evaluation  Master  Plans  (TEMP's) 
and  the  test  design  plans  (TDP’s)  for  0T8<E  of  all  systems  it 
identifies  for  oversight.  DQT8<E’s  emphasis  is  primarily  on  DOD 
major  systems;  however,  it  does  monitor  select  ADAP  and  IPR 
programs.  It  must  approve  these  plans  in  writing  prior  to 
initiation  of  operational  testing  and  by  law  DOTS<E  has  funding 
release  authority  for  the  conduct  of  these  tests.  DOTS<E 
conducts  independent  evaluations  of  these  test  results  which  it 
reports  directly  to  the  Secretary  of  Defense  and  Congress. 

Even  though  the  presence  of  DOTS<E  has  been  a  positive 
influence  in  improving  the  overall  quality  of  the  MAP,  it  is  an 
additional  impediment  to  the  timely  execution  of  individual 
service  materiel  acquisition  processes.  The  chief  criticisms 
of  DOT?<E  include  its  heavy  reliance  on  contractor  support, 
failure  of  its  staff  to  visit  t^t  sites  for  sufficiently  long 
periods  of  time  to  assess  system  performance,  and  its 
parochialism  and  strong  bias.  The  office  is  perceived  to  have 
a  preoccupation  with  and  devotes  much  of  its  efforts  to 
personalized  threat  assessment  and  sometimes  to  questionably 
required  expensive  baseline  comparison  testing.  DOTS<E  has 
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recently  completed  a  contracting  e-f-fort  -for  the  development  o-f 
an  in— house  threat  assessment  data  base. 


However,  DOT?<E’s  authority  is  substantial  and  it  has  had 
a  signi-ficant  and  positive  in-fluence  on  the  MAP.  Mr.  John  E. 
Krings,  Director  o-f  DOT?<E,  soon  a-fter  he  assumed  his  position 
as  director  stated  in  a  memiorandum  to  military  departments, 
de-fense  agencies,  and  the  National  Security  Agency; 


"Title  10,  United  States  Code  assigns  overall 
responsibility  -for  operational  test  and  evaluation 
(0T?<E)  oversight  in  the  Department  o-f  De-fense  to  the 
Director,  Operational  Test  and  Evaluation  (D0T?<E)  . 

As  ampli-fied  by  DoD  Directive  5141.2,  the  Director 
is  required  to  maintain  oversight  o-f  the  major  DoD 
programs  which  meet  the  criteria  o-f  Section  2432 
(-formerly  Section  139a)  o-f  Title  10,  together  with 
any  other  programs  he  may  choose  to  designate.... 

Programs  under  D0T8<E  oversight  are  subject  to 
the  provisions  o-f  DoD  Directive  5000.3,  "Test  and 
Evaluation."  In  particular,  -for  each  listed  program 
a  (Test  and  Evaluation  Master  Plan)  TEMP  must  be 
submitted  to  QSD  for  approval,  with  annual  updates 
to  ensure  currency,  unless  the  program  has 
progressed  past  the  point  where  appreciable  future 
testing  is  planned."  9/ 


The  new  requirement  for  a  DOD  level  review  brought  more  focus 
to  the  role  of  good  test  planning  as  part  of  a  viable 
acquisition  strategy.  From  a  negative  perspective,  this 
emphasis  has  drawn  the  Senior  Army  Staff  into  the  review  and 
approval  of  TEMP’s.  Working  documents  have  thus  become  staff 
products.  The  process  for  staffing  this  document  was  recently 
released  by  the  Deputy  Und^er  Secretary  of  the  Army  for 
Operations  Research  office.  (See  EXHIBIT  4). 
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EXHIBIT  4 


TEMP  Staffing  Process 
22  December  1988 


(1)  PEO  LEAD  -  30  DAYS  OR  LESS 

(2)  TEMA  LEAD  -  16  DAYS  OR  LESS 


Frequent  criticism  o-f  the  length  of  the  MAP  is  answered  in  part 
by  the  staffing  requirements  as  outlined  above.  The  staffing 
process  in  the  Pentagon  is  very  difficult;  it  requires  tracking 
down  people  who  are  on  travel,  briefing  and  trying  to  quickly 
educate  people  who  have  not  been  involved  with  the  program,  and 
constantly  correcting  and  retracing  steps  to  ensure  that 
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previous  reviewers  agree  with  the  newest  set  o+  corrections. 


The  amount  o-f  time  and  e-f-fort  required  to  sta-f-f  and  obtain 
approval  o-f  a  TEMP,  TDP  or  the  review  o-f  an  Independent 
Evaluation  Report  (lER)  typically  -far  exceeds  the  time  required 
to  conduct  the  test.  We  have  gone  to  great  pains  to  streamline 
the  acquisition  process,  the  sta-ffing  process  has  yet  to  be 
similarly  influenced.  Often,  staffing  requirements  become  so 
overbearing  that  over  time  they  are  either  ignored  or  are 
replaced  by  a  requirement  to  obtain  approval  from  a 
review/decision  making  board. 


Another  aspect  of  this  increased  oversight  is  explained 
by  LT  GEN  George  H.  Sylvester  (USAF,  ret.)  in  his  report  to  the 
Packard  Commission,  "The  Role  of  Test  and  Evaluation  in  Defense 
Acquisition"  (December  1985); 


"  ...Unfortunately,  the  growth  of  microscopic 
scrutiny  and  micromanagement  from  above, 
particularly  from  the  Congress  and  OSD,  have  given 
rise  to  Service  defensiveness  that  makes  it  more  and 
more  difficult  to  surface  and  come  to  grips  in  a 
timely  fashion  with  problems  unearthed  by 
testing....  Under  the  prevailing  culture,  most 
major  programs  begin  to  acquire  a  constituency  of 
their  own  soon  after  they  enter  the  development 
cycle.  In  due  course  they  take  on  a  strong  Service 
identity,  nurtured  on  the  one  hand  by  contractor 
advertising,  trade  conventions,  roll-out  ceremonies 
and  the  like  and  on  the  other  by  budget  planning, 
force  structure  planning,  IOC  dates,  etc. 

Eventually  the  program  reaches  the  status  that  it 
must  not,  indeed  cannot,  fail.  The  program  manager 
is  the  Service  agent  responsible  for  guiding  the 
system  through  the  acquisition  cycle  and  all  of  his 
incentives  are  in  the  direction  of  successfully 
deploying  the  system  on  schedule.  In  such  an 
environment,  the  independent  □T8<E  agency  (OTEA  in 
the  Army,  OPTEVFOR  in  the  Navy,  and  AFOTEC  in  the 
Air  Force)  faces  a  tough  task,  particularly  if  it  is 
to  be  the  purveyor  of  bad  news  that  their  service's 
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new  ’Golden  Albatross’  has  serious  operational 
shortcomings.  Even  the  using  command,  which  would 
stand  to  lose  the  most  -from  a  new  system  that 
performed  below  expectations,  will  more  readily 
support  the  system  as  is  than  risk  the  prospect  of 
program  delay  or  perhaps  even  cancellation  by 
raising  serious  objections.  After  all,  it  is  a  rare 
system  that  even  with  its  shortcomings  is  not  a 
significant  improvement  over  the  system  it 
replaces. . . . 

...As  a  result  of  the  powers  bestowed  upon  (the 
DOT?<E)  by  the  Congress  (and)  the  standards  of 
independence  which  the  D0T8<E  is  ex  er  c  i  si  ng  .  .  .  ther  e 
(is)  a  growing  recognition  within  the  services, 
including  the  very  senior  levels,  that  an  open, 
honest,  and  non-adversar i al  dialogue  (needs)  to  be 
established  and  maintained  on  each  Defense 
acquisition  program  from  the  very  beginning."  1 0 / 


This  new  openness  is  very  important  and  DOT?<E  is  but  one 
contributor.  Even  with  the  recent  DOD  reorganization, 
oversight  activities  are  still  in  place:  GAO,  DOD  IG,  DA  IG, 
and  the  AAA.  (See  EXHIBIT  5).  The  0T8<E  process  still  requires 
some  refinement  to  enhance  independence  and  also  to  minimize 
redundancy.  The  Secretary  of  Defense’s  role  in  the  MAP  and  TS<E 
could  be  strengthened.  Overlapping  OT&iE  Congressional 
oversight  missions  of  the  Government  Auditing  Office  with  its 
staff  of  5000+  in  offices  located  throughout  the  country  and 
D0T8<E  with  its  pitifully  small  staff  located  at  the  pentagon 
should  be  more  equitably  balanced. 
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CHAPTER  IV 


SURVIVABILITY/VULNERABILITY  LEG I SLAT I ON- 
"WHISTLE-BLOWERS" ■ 

Occasionally  the  headlines  are  filled  with  the  latest 
results  of  "investigative  reporting",  scandal  as  related  by  a 
disgruntled  former  employee.  The  "whistle-blowers"  are 
entitled  to  bounties  as  high  as  257-  of  the  damages  awarded 
which  are  offered  under  the  False  Claims  Act.  The  court 
dockets  are  growing  with  cases  of  this  type.  There  is  a 
potential  for  these  Congress! onal 1 y  mandated  awards  to  become 
subversive.  The  overall  results  are  counterproduct i ve  to  the 
MAP.  One  might  argue  that  the  savings  realized  by  the 
government  more  than  warrant  the  costs  of  rewarding  the 
source.  However,  even  individuals  working  faithfully  within 
the  government  and  trying  to  make  it  better,  are  entitled  to 
far  less.  If  they  are  lucky,  this  award  amounts  to  10  percent 
to  as  little  as  half  a  percent  of  the  amount  saved  by  the 
government.  Care  must  be  exercised  to  protect  against 
excessive  policing  and  micro  management.  Ethics,  "public 
trust",  and  responsible  actions  'Sre  very  important  and  should 
be  held  in  high  regard.  Punishment  for  indiscretions, 
violations,  or  criminal  actions  should  be  quick,  severe,  and 
proportional.  However,  cooperation  is  the  key  to  success  of 
any  organization  and  external  police  efforts  tend  to  foster 
adversarial  relationships  which  are  counterproductive. 
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Sel  f-pol  i  ci  ng  as  part  o-f  a  Total  Quality  Management  (TQM) 
program  is  far  more  preferential. 


One  of  the  most  notable  "whistle  blowers"  is  Col  Burton, 
USAF.  Col  Burton’s  criticism  was  that  new  weapon  systems  were 
vulnerable  to  live  fire.  The  implication  was  that  the  services 
were  not  subjecting  new  weapon  systems  to  actual  live  fire 
during  the  test  and  evaluation  process.  This  was  not  entirely 
true.  The  basis  for  all  operational  testing  is  typically 
categorised  into  thirteen  areas: 


Performance 

Reliability,  availability,  maintainability 
Durabi 1 i ty 
Logi sties 

Survi vabi 1 i ty/vul nerabi 1 i ty 
Mob i 1 i t  y 
Interoperabi 1 i ty 
MANPRINT: 

Human  Factors  Engineering 

Training  (system  and  tactics/doctrine) 

Safety 

Health  Hazards 

Manpower 

Personnel 


Limited  numbers  of  available  prototypes,  cost  and  time 
constraints  typically  preclude  extensive  testing  in  three  of 
these  areas:  survi vabi 1 i ty/vul nerabi 1 i ty,  interoperability, 
and  logistics.  A  key  element  in  assessing  the  effectiveness  of 
a  weapon’s  survi vabi 1 i ty/vul nerabi 1 i ty  characteristics  is  to 
determine  its  weaknesses  to  live  fire.  Actual  live  fire 
testing  tends  to  be  destructive.  Destructive  testing  is  in  fact 
an  infrequent  occurrence  for  obvious  reasons.  Prototypes  are 
typically  very  expensive,  few  in  number,  and  are  of  little 


value  after  destructive  testing.  Frequently  the  answers  with 
regard  to  systems  survivability  and  vulnerability  can  be 
estimated  more  efficiently  by  other  methods.  Estimation  of 
weapon  system  survivability  and  vulnerability  characteristics 
are  evaluated  as  a  function  of  many  factors,  such  as:  where 
the  round  hits,  the  angle  at  which  it  hits,  the  type  of  round 
the  range  at  which  the  target  is  hit,  to  name  but  a  few.  An 
infinite  number  of  combination  and  permutations  make  computer 
modeling  a  much  better  method  of  estimating  system 
survi  vabi  1  i  ty/vLil  nerabi  1  i  ty  characteristics  than  live  fire 
test i ng . 


However,  if  the  intent  of  Congress  was  to  place  more 
emphasis  and  attention  on  this  important  area  of  assessment, 
they  have  accomplished  their  end.  In  a  letter  to  the 
Secretaries  of  the  Military  Departments  (dated  25  November 
1986)  Mr.  Godwin,  the  Defense  Acquisition  Executive  (DAE), 
offers  a  reasonable  and  logical  interpretation  of  this  new 
survi vabi 1 i ty/vul nerabi 1 i ty  1 egi si ati on: 

"The  spirit  of  this  new  legislation  requiring  Live 
Fire  Testing  is  to  cause  weapons  designers  and 
proponents  to  assess  the  survivability  and  lethality 
of  their  systems  sufficiently  early  in  the 
development  process  to  permit  effective  and 
efficient  fixes  in  a  timely  manner.  The  overall 
objectives  of  Live  Fire  Testing  are: 

-  Estimation  of  vulnerability  of  combat-configured 
system  to  threat  munitions, 

-  Estimation  of  lethality  of  munitions  against 
their  principle  foreign  targets,  and 

-  Provision  of  early  insights  into  design  changes 
or  modifications  necessary  to  reduce  vulnerabilities 
and/or  improve  lethality."  11/ 


Both  the  U.S.  Army  Ballistic  Research  Laboratory  (BRL) 


and  U.S.  Army  Materiel  Systems  Analysis  Agency  (AMSAA)  are 
involved  in  the  assessment  and  evaluation  of  the 
SLirvi  vabi  1  i  ty  /  VLil  nerab  i  1  i  ty  per-formance  characteristic  o-f  new 
weapon  systems.  Test  and  Evaluation  Command  (TECDM)  -frequently 
conducts  limited  live  fire  testing.  The  data  from  these  tests 
are  used  to  supplement  extensive  computer  modeling  and 
simulation  studies  of  live  fire  effects.  Results  from  these 
lethality  models  are  then  used  to  evaluate  the  effectiveness  of 
weapon  system  design.  A  potential  weakness  here  is  that  both 
BRL  and  AMSAA  should  play  a  more  active  role  earlier  in  the  MAP 
and  assist  (1)  program  managers  in  drafting  realistic 
survi vabi 1 i ty/vul nerabi 1 i ty  specifications  and  (2)  contractors 
in  the  development  of  optimum  survi vabi 1 i ty/vul nerabi 1 i ty 
design.  Department  of  Defense  has  devoted  an  extensive  amount 
of  money  and  resources  to  the  study  of  live  fire  effects.  In 
March  1984  the  Office  of  the  Director,  Defense  Test  and 
Evaluation  Office  chartered  the  Joint  Live  Fire  Program  (JLF) . 
This  program  has  been  very  successful. 


"Over  the  last  three  years,  these  JLF  efforts  have 
yielded  numerous  insights  into  tTie  way  our  combat  systems 
would  actually  respond  in  combat  situations.  Testing 
continues  under  this  program  at  Wright  Patterson  AFB,  Naval 
Weapons  Center  (China  Lake),  Edwards  AFB,  Aberdeen  Proving 
Ground,  Soccoro  NM,  and  other  test  facilities.  U.S.  and 
foreign  munitions  and  weapons  platforms  (tanks,  armored 
personnel  carriers,  fixed  and  rotary  wing  aircraft, 
antitank  guided  munitions,  shoulder  launched  shaped  charge 
munitions,  mines,  kinetic  energy  penetrators  as  well  as  a 
host  of  other  systems)  have  been,  and  continue  to  be, 
tested  within  the  JLF  under  DOD  sponsorship,  examining 
systems  that  have  already  arrived  on  the  battlefield  in 
significant  numbers."  12/ 
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The  question  that  remains  is  should  live  -fire  testing  be  more 
e>!  tensi  ve? 


In  response  to 
mandated  the  answer  to 
merits  o-f  this  testing 
Military  Authorisation 
weapon  systems: 


Col  Burton’s  revelations,  Congress 
this  question.  Instead  o-f  assess!  n 
on  a  case  by  case  basis,  the  FYS? 
Act  requires  the  following  for  all 


g  the 


ma  j  or 


"(a)  REQUIREMENTS. — (1)  The  Secretary  of  Defense 
shall  provide  that — 

(A)  a  covered  system  may  not  proceed  beyond 
low-rate  initial  production  until  realistic 
survivability  testing  of  the  system  is 
completed . . . . ; 

<B)  a  major  munition  program  or  a  missile 
program  may  not  proceed  beyond  low-rate  initial 
production  until  realistic  lethality  testing  of  the 
program  is  complete....;  and 

(C)  a  major  defense  acquisition  program  may 
not  proceed  beyond  low-rate  initial  production 
until  initial  operational  test  and  evaluation  of 

the  program  is  completed . 

(b)  GUIDELINES- — (1)  Survivability  and  lethality 
tests  required  under  subsection  (a)  shall  be 
carried  out  sufficiently  early  in  the  developmental 
phase  of  the  system  or  program  (including  a  covered 
product  improvement  program  to  allow  any  design 
deficiency  demonstrated  by  the  testing  to  be 
corrected  in  the  design  of  the  system,  munition,  or 
missile  (or  in  the  product  modification  or  upgrade 
to  the  system,  munition,  or  missile)  before 
proceeding  beyond  low-rate  initial  production." 

13/ 


The  problem  here  again  is  that  "sufficiently  early"  is  too 
late.  AMSAA  and  BRL  need  to  become  part  of  the 
survi vabi 1 i ty/vul nerabi 1 i ty  design  efforts  during  the 
pre-prototyping  design  phase. 
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Further,  in  a  time  where  resources  are  severely 


limited,  testing  is  not  always  cost  e-f-fective  nor  warranted. 
"Testing— for-testing-sake"  is  irresponsible  when  results  can  be 
obtained  -from  other  sources.  There  are  systems  which  are  "so-ft 
targets",  trade-o-f-fs  have  been  openly  made  -for  the  sake  o-f 
other  higher  priority  per-formance  characteristics,  such  as 
mobility,  combat  weight  restrictions,  MANPRINT,  and  others.  A 
completely  invulnerable  weapon  system  has  yet  to  be  built, 
legislative  micro-management  dictating  wholesale 
survi vabi 1 i ty/vul nerabi 1 i ty  testing  is  in  error  here.  A 
Congressional  vehicle  -for  assuring  adequate 

survi vabi 1 i ty/vul nerabi 1 i ty  test  and  evaluation  is  already  in 
place.  The  De-fense  Operational  Testing  and  Evaluation  (DQT?<E) 
0-f-fice  is  charged  to  ensure  that  operational  testing  is 
adequate  and  to  provide  status  reports  to  Congress.  Why 
legislate  or  rather  dictate  live  -fire  testing  across  the  board 
when  it  may  not  always  be  appropriate.  DQT?<E  can  service  this 
Congressional  management  concern  much  more  e-ffectively  and  with 
-far  less  criticism.  Further  a  key  -factor  in  adequate  live  fire 
testing  is  identifying  the  proper  threat.  DOTS<E  maintains  a 
current  threat  database  and  is  iTi  an  excellent  position  to 
ensure  that  baseline  threat  questions  are  assessed  and 
answered . 
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CHAPTER  V 


CONTRACTOR  PARTICIPATION  LEGISLATION- 
GOVERNMENT  AUDIT  ORGANIZATION. 

Operational  testers  have  always  been  very  sensitive  to 
contractor  in-fluence  or  even  perceptions  o-f  contractor 
influence  on  operational  testing.  As  a  matter  of  policy 
contractors  have  been  allowed  to  observe  pilot  testing,  but 
once  operational  testing  has  begun,  are  precluded  from 
participating  in  record  testing.  Record  testing  is  individual 
system  test  time  when  realistic  operational  data  is  being 
recorded  for  use  to  support  the  operational  evaluator’s 
analysis  and  report.  This  subtle  qualification  is  necessary 
because  contractors  frequently  do  play  a  legitimate  role  in 
operational  test  administration  by  representing  higher  levels 
of  maintenance  (i.e.,  direct  support,  general  support,  depot, 
and  contractor).  This  support  is  provided  as  follows:  The 
contractor  trains  a  government  training  team  on  the  system. 

The  government  training  team  trains  military  to  be  system 
operators  and  uni t/organi zati on  level  maintainers.  When  a 
system  has  a  maintenance  probl  en>»  dur  i  ng  testing,  military 
maintainers  attempt  to  correct  the  problem.  The  adequacy  of 
their  training,  manuals,  and  test  equipment  are  evaluated.  If 
the  military  maintainers  fail  to  correct  the  problem  and  all 
other  maintenance  alternatives  are  exhausted,  contractor 
maintenance  is  requested.  Contractor  activities  are  closely 
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monitored;  all  parts  used,  maintenance  times,  and  other  data 
are  -fully  documented.  When  the  problem(5)  has  been  corrected, 
the  contractor  leaves  the  test  area  and  the  test  item  resumes 
record  testing.  One  might  rationalize  this  limited  role  of  the 
contractor  by  (1)  there  are  always  unforeseen  maintenance 
problems  or  design  flaws  that  fall  outside  the  scope  of 
preliminary  maintenance  training  plans.  Without  limited, 
controlled  contractor  maintenance  support,  testing  would 
quickly  come  to  a  halt  precluding  collection  of  further 
performance  and  maintenance  data.  (2)  Current  system 
maintenance  concepts  call  for  built-  in-test  and  diagnostic 
equipment  (BITE)  and  modular  replacement  at  the 

uni t /organ! 2 at i on  levels  of  maintenance.  Rather  than  trying  to 
repair  defective  parts  in  the  field,  they  are  evacuated  to 
higher  levels  of  maintenance.  Many  of  these  higher  level 
maintenance  activities  have  been  centralized  and  ci vi 1 i ani zed . 
And  (3)  currently  many  major  systems  have  maintenance  concepts 
which  call  for  contractor  maintenance  support  of  the  fielded 
system.  There  are  two  other  roles  that  contractors  have  played 
in  operational  testing  but  these  are  less  frequent:  providing 
technical  clarification  to  the  RAM  Scoring  Conference 
membership  and  providing  complex’  data  collection 
instrumentation  and  automation  and  data  verification  support. 

As  part  of  the  T?<E  process,  meetings  are  held  during 
and  just  after  both  developmental  and  operational  tests  to 
review  reliability,  availability  and  maintainability  (RAM) 
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data,  RAM  Scoring  Con-f erences.  Data  is  collected  during  tests 
to  record  all  the  activities  and  actions  associated  with  system 
maintenance  problems.  This  data  includes  the  time  that  it 
takes  to  correct  the  problem,  the  manhours  o-f  maintenance  time 
required  at  each  level  o-f  maintenance,  the  impact  of  the 
problem/failure  on  the  ability  of  the  system  to  perform  its 
mission,  and  a  wealth  of  other  valuable  data.  A  scoring 
conference  committee  reviews  each  RAM  incident  to  determine  the 
probable  cause  (i.e.,  hardware,  software,  crew,  training,  etc.) 
and  type  of  failure  (i.e.,  operational  mission  failure,  system 
failure,  etc.).  The  basis  for  this 

chargeabi 1 i ty /cl assi f i cat i on  process  is  a  formal  set  of  rules 
documented  in  the  approved  Failure  Definition/Scoring  Criteria 
for  the  system.  At  the  end  of  each  test  these  scoring 
conferences  are  fallowed  by  a  final  RAM  Assessment  Conference. 
RAM  incidents  are  reviewed  in  terms  of  grouping  multiple 
failures  or  related  failures,  types  and  severity  of  failures, 
training  problems,  assessing  proposed  corrective  actions  or  the 
effectiveness  of  previously  applied  corrective  actions.  The 
purpose  of  this  complex  process  is  to  thoroughly  assess  RAM 
data  and  to  develop  a  common  RAM  data  base  to  support  the  RAM 
characteristic  evaluation  of  a  s'ystem  by  all  parties 
concerned . 

As  valuable  and  rational  as  this  process  seems,  one  of 
its  major  weaknesses  is  the  limited  expertise  of  the  membership 
of  these  RAM  conferences.  As  weapon  systems  become  more 


complex  and  civil  service  pro-f  essi  onal  engineering  salaries 
become  increasingly  less  competitive,  hiring  and  ret^iining 
quali-fied  engineering  personnel  becomes  increasingly  more 
di-f-ficLilt.  Therefore,  contractor  participation  in  RAM  scoring 
and  assessment  conferences  is  frequently  required  to  address 
questions  regarding  probable  causes  of  failures  and  related 
complex  weapon  system  design  questions.  Unfortunately,  in  the 
interest  of  obtaining  a  comprehensive  understanding  of  the  RAM 
characteristics  of  the  weapon  system,  there  is  a  significant 
potential  for  undue  contractor  influence  on  the  RAM  scoring 
process.  The  seriousness  of  this  RAM  scoring  problem  was  cited 
by  the  GAO.  The  potential  for  contractors  to  unfairly  bias 
independent  test  results  was  deemed  unacceptable  and  placed  the 
credibility  of  operational  testing  in  jeopardy. 

Contractors  may  play  other  roles  in  OT.  On  occasion, 
contractors  develop  complex  diagnostic  equipment  to  assist  them 
in  the  development  of  very  complex  systems.  This  equipment  can 
provide  valuable  insights  into  the  probable  cause  of  system 
failures  that  occur  during  OT  and  can  greatly  assist  in 
resolving  data  collection  anomalies.  Frequently  this 
instrumentation  is  the  only  reliable  source  of  data  which  can 
be  used  to  at  least  partially  evaluate  critical  elements  and 
subelements  of  performance.  Without  access  to  this  data 
source,  evaluators  would  be  forced  to  severely  limit  their 
assessments  and  to  present  their  findings  as  test  limitations 
(i.e.,  data  not  available  to  answer  the  questions).  The 


complexities  o-f  this  instrumentation  and  associated  automated 
data  reduction  requirements  frequently  require  the  use  of 
contractor  technical  personnel,  ADP  support  equipment,  and 
facilities.  Also,  use  of  this  instrumentation  can  greatly 
assist  the  contractor  in  the  isolation  of  the  cause (s)  of 
failures  observed  during  testing  and  more  efficiently  redesign 
the  system  to  correct  these  problems  and  to  produce  a  system 
that  meets  the  Army’s  needs.  Strict  prohibition  of  contractor 
involvement  in  operational  testing  would  preclude  the  evaluator 
from  more  fully  assessing  system  performance. 


In  response,  Congress  included  the  following  in  the 
FYS?  Military  Authorization  Act: 


"  <2)  In  the  case  of  a  major  defense 

acquisition  program,  no  person  employed  by  the 
contractor  for  the  system  being  tested  may  be 
involved  in  the  conduct  of  the  operational  test  and 
evaluation  required  under  subsection  (a).  The 
limitation  in  the  preceding  sentence  does  not  apply 
to  the  extent  that  the  Secretary  of  Defense  plans 
for  persons  employed  by  that  contractor  to  be 
involved  in  the  operation,  maintenance,  and  support 
of  the  system  being  tested  when  the  system  is 
deployed  in  combat.’’  14/ 


The  intent  of  Congress  in  passing  this  legislation  was  the 
subject  of  considerable  debate  arniong  the  service  OTS^E  groups, 
and  services’  interpretation  of  the  law  vary  widely. 

Currently,  there  is  no  formal,  joint  coordination  mechanism 
between  the  services  judge  advocates  for  the  interpretation  of 


publ i c  1  aw 


Even  though  Congress’s  motives  in  passing  this  law  were 
based  upon  several  isolated  incidents,  an  interpretation  o-f  the 


law  would  indicate  that  Congress  was  taliing  broad  action 
without  considering  the  costs  and  consequences.  The  Army, 
which  tends  to  be  more  literal  in  its  i  nterpretat  i  on  o-f 
legislation  than  the  other  services,  proved  to  be  right  in  this 
case.  Mr,  B.  Goldwater,  Chairman  o-f  the  Senate  Committee  on 
Armed  Services,  in  a  5  Jan  87  letter  to  Mr.  C.  Weinberger, 
Secretary  o-f  De-fense,  provided  the  -following: 


"  ...It  is  our  intention  to  ensure  that,  during 

operational  tests,  weapon  systems  are  operated, 
maintained,  and  otherwise  supported  by  personnel 
typical  o-f  those  who  carry  out  such  duties  when  the 
system  is  deployed  in  combat.  It  is  our  -further 
intention  that  the  processing  and  evaluation  o-f  test 
data  be  carried  out  in  a  completely  objective  manner 
with  no  possibility  or  even  the  appearance  of 
system-contractor  manipulation. 

There-fore,  under  the  limitation  established  by 
PL  99-661,  system-contractor  personnel  may  be 
involved  in  an  operational  test  only  to  the  extent 
that  is  planned  -for  them  to  be  involved  in  the 
operation,  maintenance,  and  other  support  o-f  the 
system  being  tested  when  that  system  is  deployed  in 
combat.  We  would  expect  that  very  -few,  i -f  any, 
conventional  combat  systems  as  de-fined  in  PL  99-661 
will  entail  such  post-development  contractor 
involvement.  Thus,  -few,  it  ^"iny,  operational  tests 
will  entail  any  sort  o-f  system— contractor 
participation."  15/ 


An  attempt  by  Mr.  Krings,  director  o-f  DDTS<E,  to  convince 
Congress  to  allow  the  services  more  latitude  in  controlling 
contractor  involvement  in  operational  testing  met  with 
resistance.  In  a  31  May  1988  article  in  "De-fense  Week"  Mr. 
Joseph  Cirincione,  the  Congressional  committee  staff  member  who 
drafted  the  legislation  stated: 
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"There's  a  real  good  reason  why  you  don't  want 
contractors  involved  in  the  tests,...  The  systems 
should  be  tested  like  they  will  be  -fielded  in  combat 
without  contractor  support... 

It  seems  silly  to  try  to  change  the  law 
agai n ....  Congress  has  tried  to  get  tough  on 
operational  testing  while  Krings'  o-f-fice  has  tried 
to  make  it  more  bureaucratically  efficient."  16/ 

Contrary  to  popular  belief,  several  major  weapon  systems 
have  wartime  contractor  maintenance  support  concepts.  However, 
the  problems  associated  with  eliminating  all  contractor  support 
from  testing  are  far  greater.  A  case  in  point,  competitive 
testing  as  part  of  source  selection  processes  is  a  common 
occurrence.  The  time,  cost  and  negative  training  impacts  of 
training  soldiers  to  perform  all  levels  of  maintenance  for  all 
competing  systems  is  ludicrous.  High  level  maintenance 
training  typically  requires  months  of  classroom  and  hands-on 
system  time.  Further,  the  frequent  occurrence  of  unanticipated 
system  failures,  the  maintenance  for  which  may  have  not  been 
included  in  the  preliminary  training  package,  would  result  in 
costly  and  unwarranted  termination  of  nearly  every  operational 
test  conducted.  However,  contractor  participation  in  the  RAM 
process  is  perhaps  unwarranted;  the  merits  of  other  limited 
contractor  participation  in  the  conduct  of  operational  testing 
should  be  assessed  on  a  case  by  case  basis  and  should  be 
strictly  controlled. 
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CHAPTER  VI 


CONCLUSIONS 

Department  o-f  the  Army’s  Materiel  Acquisition  Process 
has  been  re-fined  over  a  period  o-f  many  years.  The  Army’s 
commitment  to  the  continued  imp'^ovement  o-f  the  MAP  in  order  to 
better  facilitate  the  acquisition  of  the  weapon  system,  to 
include  all  associated  support  documentation,  training, 
facilities,  and  equipment  is  long  standing.  The  inherent 
complexities  of  developing  and  fielding  a  complete  weapon 
system  package  are  often  overlooked  when  critics  rate  the 
effectiveness  of  the  MAP.  The  Army’s  MAP  is  a  comprehensive, 
efficient,  and  effective  process  when  materiel  developers 
adhered  to  it.  However,  no  matter  how  good  the  process  is, 
without  effective  and  dynamic  management,  weapon  system 
materiel  acquisition  efforts  will  have  far  less  chance  of 
success.  Significant  improvements  can  still  be  made  in  the 
MAP.  A  major  consideration  among  actions  which  would  make  this 
possible  is  that  we  need  to  put  the  "T"  back  into  □T?<E.  More 
emphasis  needs  to  be  placed  on  the  quality  and  independence  of 
operational  testing  as  well  as  operational  evaluation. 

Further,  DOTS<E  needs  to  shed  itself  of  much  of  its  contractor 
support,  assume  more  comprehensive  OTS<E  responsibility,  and 
increase  its  personnel  strength.  If  Congress  is  serious  about 
having  a  credible  DOT&E  office,  this  office  should  be  held 
accountable  for  ensuring  that  survivability/vulnerability 
testing  is  adequate  and  that  the  level  of  contractor 
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part i ci pati on  in  operational  testing  does  not  adversely  impact 
on  the  credibility  ot  the  testing  e-f-forts.  To  preclude 
duplication  o-f  e-f-fort,  DOTS<E  should  -fully  assume  the  miission 
and  assimilate  a  significant  portion  of  the  GAO  assets  that  cire 
currently  dedicated  to  the  review  of  the  MAP  for  Congress.  In 
summary,  isolated  management  problems  are  frequently  cited  as 
e;-;amples  of  the  failure  of  the  MAP.  Reorganization,  broad 
policy  changes,  and  more  recently  legislative  actions  have 
typically  been  offered  as  the  solutions  to  these  problems. 

These  actions  while  correcting  one  perceived  problem  frequently 
cause  other  more  serious  problems.  Overall  the  MAP  is  a  proven 
and  very  effective  approach;  refinement  and  more  attention  to 
total  quality  management  are  the  real  answers  to  improving  this 
critical  process  which  governs  the  engineering  development  and 
fielding  of  all  Army  systems. 
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CHAPTER  VII 


RECQMMENDATIQNS 


*  In  the  interest  o-f  circumventing  the  perception  o-f 
conflict  of  interest,  members  of  the  House  or  the  Senate 
Appropriations,  Authorization,  Budget,  and  Armed  Services 
Committees  should  be  prohibited  from  either  directly  or 
indirectly  receiving  campaign  contributions,  consultant  fees 
and/or  honoraria  from  defense  contractors  and/or 
subcontractors. 


*  Department  of  the  Army  (and  each  service)  should 
establish  an  operational  (1)  test  and  (2)  evaluation  activity 
(or  two  complementary  activities)  responsible  for  the  conduct 
al  1  Army  system  acquisition  OTS<E.  This  organization  must  be 
completely  independent  of  the  chain  of  command  of  the  combat 
developer,  the  materiel  developer,  or  the  program  manager.  An 
independent  Army  OT&E  command  reporting  directly  the  the  CSA 
would  be  a  major  improvement.  (Centralization  of  mul t i -servi ce 
OT?<E  functions  at  the  DQD  level  is  not  recommended.  This  would 
result  in  a  significant  loss  of  independence  and  would  not 
serve  the  fair  and,  at  least  initially,  uncompromising 
consideration  of  many  of  the  service  unique  requirements  of  the 
American  fighting  man;  soldier,  sailor,  and  aviator.) 
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*  Improvement  and  more  stringent  internal  management  o-f 
the  MAP  is  warranted.  Giving  "whi  st  1  e-bl  owers "  a  bounty  -far 
identifying  problems  is  wrong  and  the  practice  should  be 
ter mi nated . 


*  AMSAA  and  BRL  should  assist  the  program  manager  in 
drafting  realistic  survivability/vulnerability  system 
specifications  and  play  an  active  role  in  assisting  contractors 
in  optimizing  survi vabi 1 i ty/vul nerabi 1 i ty  characteristics 
particularly  during  the  pre-prototype  design  phase. 


*  The  scope  of  survi vabi 1 i ty/vul nerab i 1 i ty  "live  fire" 
testing  should  be  limited  and  used  to  verify  computer 
simulation  estimates  of  system  survi vabi 1 i ty/vul nerabi 1 i ty 
characteristics.  This  testing  and  analysis  should  be  closely 
managed  and  should  be  under  the  close  oversight  of  DOT?<E. 


*  System  contractor /subcontractor  participation  in 
operational  test  RAM  Scoring  and* Assessment  Conferences  is 
unwarranted.  Limited  contractor /subcontractor  participation  in 
operational  non-record  testing  activities,  such  as,  providing 
essential  higher  level  maintenance  support,  sophisticated  test 
instrumentation,  and  other  technical  support  are  occasionally 
warranted  and  should  be  closely  managed  by  OTEA  and  monitored 


by  DOTS<E. 


*  The  DQT?<E  organ!  sat  i  on  needs  to  be  ree;iamined  and 
cons!  derat  i  on  should  be  given  to  maintaining  a  sta-f-f  element  at 
the  Pentagon  with  a  -field  operating  activity  located  elsewhere, 

*  The  overlapping  MAP  oversight  and  investigative 
missions  o-f  the  GAO,  DQD  IG,  D0T8<E,  DA  IG,  and  AAA  need  to  be 
examined  and  resources  shi-fted  to  support  the  most  e-ffective 
accomplishment  o-f  these  important  missions. 

*  Permissible  review  time  allocated  for  staffing  of 
TEMP’S  should  be  strictly  controlled.  Failure  to  respond 
should  be  assumed  concurrence.  The  amount  of  required 
coordination  should  be  strictly  limited.  The  creation  of  a 
DQD/DA  level  TEMP  Review  and  Approval  Board  (TRAB)  would 
expedite  the  process  even  further. 

*  ”Test-f or-testing  sak^'  should  oe  closely  guarded 
against.  Requirements  for  expensive  baseline  system  comparison 
testing  should  be  fully  thought  out  and  closely  monitored  by 
DOT?cE. 
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H(  The  ability  o-f  contractor  to  provide  wartime 


maintenance  support  should  be  reexamined.  Current  civilian 
technical  support  contractual  agreements  during  period  o-f  war 
are  tenuous  at  best. 

$  D0T8<E  should  oversee  the  early  identification  of  the 
baseline  threat  which  will  be  used  as  the  basis  for  operational 
testing  and  which  should  be  included  in  the  TEMP.  DOTS/.E 
currently  maintain  an  in-house  threat  database  to  assist  them 
in  this  effort. 


*  A  viable  methodology  to  support  the  0T8<E  of  system 
logistics  characteristics  is  critically  needed. 


*  The  QT&E  of  system  interoperability  characteristics 
should  be  refined  and  expanded.  A  serious  commitment  to  the 
funding  and  establishment  of  "te%t  beds"  will  be  required  to 
make  the  necessary  improvements  in  this  important  test  area. 
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*  Program  Manager  Acquisition  Strategy  approval 
processes  should  be  re-examined.  Requirements  -for  substantial 
engineering  development  are,  at  times,  down  played  in  order  to 
-facilitate  the  pursuit  o-f  a  Non-Devel  opmental  Item  (NDI) 
strategy.  I-f  we  are  going  to  have  a  MAP,  it  should  be  -followed. 

)(c  Force  Development  Testing  and  the  development  o-f  an 
approved  TRADOC  system  specific  tactics  document  (which 
exploits  the  full  capabilities  of  the  new  system)  should  be  an 
established  prerequisites  for  the  start  of  DT. 

!K  Operational  Assessments  for  major  weapon  systems 
should  be  conducted  early  in  the  MAP  at  the  contractor’s  plant 
(if  necessary)  to  facilitate  the  timely  correction  of  problems 
and  the  exchange  of  military  expertise.  This  requirement 
should  be  strictly  enforced  and  be  a  condition  for  entry  into 
the  full  scale  engineering  devel opment /devel opment-product i on 
prove  out  phase  of  the  MAP  for  major  weapon  systems. 

*  A  formal  system  and  oversight  responsibilities  for 
tracking  the  correction  of  system  problems  discovered  during 
testing  need  to  be  established  and  institutionalized.  RAM  and 
MANPRINT  T&E  represent  the  greatest  sources  of  these  problems. 


*  Establishment  o-f  a  comprehensi  ve ,  centrally  managed 
database  containing  discrete  system  and  -funding  in-formation  tor 
all  systems  in  the  MAP  should  be  a  major  priority  in  re-fining 
the  MAP. 

*  A  joint  or  mul t i -servi ce  coordination  process  should 
be  established  by  the  -four  services'  Adjutant  Generals  to 
-facilitate  improved  and  consistent  compliance  with  new  service 
related  Congressional  legislation. 
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